{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.9 多层感知机的从零开始实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "1.6.0\n"
    }
   ],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import sys\n",
    "sys.path.append(\"..\") # 为了导入上层目录的d2lzh_pytorch\n",
    "import d2lzh_pytorch as d2l\n",
    "\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.9.1 获取和读取数据\n",
    "这里继续使用Fashion-MNIST数据集。我们将使用多层感知机对图像进行分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.9.2 定义模型参数\n",
    "\n",
    "给定一个小批量样本$\\boldsymbol{X} \\in \\mathbb{R}^{n \\times d}$，其批量大小为$n$，输入特征数为$d$。假设多层感知机只有一个隐藏层，其中隐藏单元个数为$h$。记隐藏层的输出（也称为隐藏层变量或隐藏变量）为$\\boldsymbol{H}$，有$\\boldsymbol{H} \\in \\mathbb{R}^{n \\times h}$。因为隐藏层和输出层均是全连接层，可以设隐藏层的权重参数和偏差参数分别为$\\boldsymbol{W}_h \\in \\mathbb{R}^{d \\times h}$和 $\\boldsymbol{b}_h \\in \\mathbb{R}^{1 \\times h}$，输出层的权重和偏差参数分别为$\\boldsymbol{W}_o \\in \\mathbb{R}^{h \\times q}$和$\\boldsymbol{b}_o \\in \\mathbb{R}^{1 \\times q}$\n",
    "\n",
    "其输出$\\boldsymbol{O} \\in \\mathbb{R}^{n \\times q}$的计算为\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\boldsymbol{H} &= \\boldsymbol{X} \\boldsymbol{W}_h + \\boldsymbol{b}_h,\\\\\n",
    "\\boldsymbol{O} &= \\boldsymbol{H} \\boldsymbol{W}_o + \\boldsymbol{b}_o,\n",
    "\\end{aligned}      \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们在3.6节（softmax回归的从零开始实现）里已经介绍了，Fashion-MNIST数据集中图像形状为 $28 \\times 28$，类别数为10。本节中我们依然使用长度为 $28 \\times 28 = 784$ 的向量表示每一张图像。因此，输入特征数 `d` 为784，输出个数 `q` 为10。\n",
    "\n",
    "首先，我们将实现一个具有一个隐藏层和256个隐藏单元的MLP。即 `h`=256；请注意，我们可以将这两个量都视为超参数。通常，我们选择2的幂的层宽度，由于在硬件中如何分配和寻址内存，因此在计算上趋于高效。\n",
    "\n",
    "同样，我们将用几个张量表示参数。请注意， 对于每一层，我们必须跟踪一个权重矩阵和一个偏差矢量。与往常一样，我们为这些参数的损耗梯度分配内存\n",
    "\n",
    "d = 784 ; h=256; q = 10; \n",
    "\n",
    "$\\boldsymbol{W}_h \\in \\mathbb{R}^{d \\times h}$   、$\\boldsymbol{b}_o \\in \\mathbb{R}^{1 \\times q}$  、$\\boldsymbol{W}_o \\in \\mathbb{R}^{h \\times q}$  、$\\boldsymbol{b}_o \\in \\mathbb{R}^{1 \\times q}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_inputs, num_outputs, num_hiddens = 784, 10, 256 \n",
    "# 一个隐藏层有256个隐藏单元，而不是256个隐藏层\n",
    "\n",
    "W1 = torch.normal(0, 0.01, (num_inputs, num_hiddens), dtype=torch.float, requires_grad=True)\n",
    "b1 = torch.zeros(num_hiddens, dtype=torch.float, requires_grad=True)\n",
    "W2 = torch.normal(0, 0.01, (num_hiddens, num_outputs), dtype=torch.float, requires_grad=True)\n",
    "b2 = torch.zeros(num_outputs, dtype=torch.float, requires_grad=True)\n",
    "\n",
    "params = [W1, b1, W2, b2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.9.3 定义激活函数\n",
    "\n",
    "这里我们使用基础的`max`函数来实现ReLU，而非直接调用`relu`函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def relu(X):\n",
    "    #2个参数，dtype必须相同\n",
    "    # input 和 other 的形状不需要匹配，有广播机制\n",
    "    # other = torch.zeros_like(X)\n",
    "    return torch.max(input=X, other=torch.tensor(0.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.9.4 定义模型\n",
    "\n",
    "同softmax回归一样，我们通过`view`函数将每张原始图像改成长度为`num_inputs`的向量。然后我们实现上一节中多层感知机的计算表达式\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "\\boldsymbol{H} &= \\phi(\\boldsymbol{X} \\boldsymbol{W}_h + \\boldsymbol{b}_h),\\\\\n",
    "\\boldsymbol{O} &= \\boldsymbol{H} \\boldsymbol{W}_o + \\boldsymbol{b}_o,\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def net(X):\n",
    "    X = X.view((-1, num_inputs))\n",
    "    Y = torch.matmul(X, W1) + b1\n",
    "    H = relu(Y)\n",
    "    O = torch.matmul(H, W2) + b2\n",
    "    return O"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.9.5 定义损失函数\n",
    "\n",
    "为了得到更好的数值稳定性，我们直接使用PyTorch提供的包括softmax运算和交叉熵损失计算的函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = torch.nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.9.6 训练模型\n",
    "\n",
    "训练多层感知机的步骤和3.6节中训练softmax回归的步骤没什么区别。我们直接调用`d2lzh_pytorch`包中的`train_ch3`函数，它的实现已经在3.6节里介绍过。我们在这里设超参数迭代周期数为5，学习率为100.0。\n",
    "> 注：由于原书的mxnet中的`SoftmaxCrossEntropyLoss`在反向传播的时候相对于沿batch维求和了，而PyTorch默认的是求平均，所以用PyTorch计算得到的loss比mxnet小很多（大概是maxnet计算得到的1/batch_size这个量级），所以反向传播得到的梯度也小很多，所以为了得到差不多的学习效果，我们把学习率调得成原书的约batch_size倍，原书的学习率为0.5，这里设置成100.0。(之所以这么大，应该是因为d2lzh_pytorch里面的sgd函数在更新的时候除以了batch_size，其实PyTorch在计算loss的时候已经除过一次了，sgd这里应该不用除了)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "epoch 1, loss 0.0030, train acc 0.713, test acc 0.792\nepoch 2, loss 0.0019, train acc 0.823, test acc 0.816\nepoch 3, loss 0.0016, train acc 0.846, test acc 0.794\nepoch 4, loss 0.0016, train acc 0.853, test acc 0.839\nepoch 5, loss 0.0015, train acc 0.864, test acc 0.830\n"
    }
   ],
   "source": [
    "num_epochs, lr = 5, 100.0\n",
    "d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, params, lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.7 64-bit ('d2l': conda)",
   "language": "python",
   "name": "python37764bitd2lconda94fc7ab78ae34cabbef0e75f5636f253"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}